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Symposium 

Evaluating Program Effectiveness 
in the Safe Schools Healthy 
Students Initiative 



Symposium Introduction 

Oliver T. Massey 

The Safe Schools/Healthy Students Initiative (SS/HSI) is a federal 
grant to school districts intended to promote health and safety in 
schools and communities. The Initiative is an ambitious attempt to 
address the increased recognition of violence and threats to the safety 
and welfare of students in our nations schools. The Initiative was 
designed to confront school violence, safety and mental health needs 
through the provision of a broad array of prevention and intervention 
services. While partly predicated by the tragedies of Columbine and other violent school incidents, the 
far more common and perhaps mundane concerns of bullying, teasing, and respect for authority also 
drove major programmatic elements of the grant. 

The programs that emerged from this grant initiative incorporated unique local solutions to vexing 
problems confronting schools. Service elements included prevention, direct interventions, and 
educational efforts. Services were to be provided by the school district, community service agencies, 
police and sheriffs departments, and other elements of the public service system. In all cases, 
programs selected for implementation were to be of proven effectiveness or identified as promising but 
of not yet established utility. The combinations of service arrays eventually implemented around the 
nation proved to be as unique as the individual demographics and needs of the schools they served. 

An additional outstanding feature of the Initiative was the mandatory set aside of a minimum of 
five percent of the budget for evaluation of grant activities. The evaluation was to provide both 
formative feedback to community stakeholders, an evaluation of the outcomes of local programs, and 
contribute to the national evaluation of the Safe Schools effort. Thus, from the evaluators perspective, 
the Initiative required the integration of multiple evaluation perspectives in addressing a broad array of 
policy, prevention, and intervention efforts with varying ties to traditional school district activities. 

The presentations included in this symposium illustrate the nature of school based interventions 
and corresponding evaluation efforts. The presentations include evidence of the unique approaches 
used to engage service providers, implement programs, and accomplish evaluation activities. The 
papers also provide evidence of the effectiveness of the interventions in the local setting with 
implications for the transportability of the programs to other schools. These efforts demonstrate not 
only the importance of mental health services in schools, but also evaluation techniques that can be 
used to establish the value of services for our students. 

In summary, the Safe Schools/Healthy Students Initiative brought together a wide range of school 
and community based partners for the provision of services aimed at youth violence, mental health, 
and safety in schools. The challenge in each site was to identify and develop a cogent plan of services 
and evaluation that took into account the interests, experience and cooperation of school and 
community program teams given the resources and unique opportunities that existed in the school 
district. Activities should integrate evaluation into a coherent plan that captures activities of the school 
district, demonstrates fulfillment of grant obligations, and documents the effectiveness of individual 
programs. The success of these efforts is well demonstrated in the papers that follow. 
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The Impact of Coordinated Community Mental Health 
Services: Safe Schools/Healthy Students 

Sharon Telleen, Young 0. Rhee Kim, Helen Stewart-Nava, & Susan Maher 

Introduction 

Provision of community mental health services for children with violent, aggressive behavior is one 
of the six elements of the Safe Schools/Healthy Students Initiative, funded by the National Institute of 
Mental Health, US Department of Education and US Department of Justice. Using a community- 
based intervention in the system of care (Duchnowski, Kutash, &c Friedman, 2002) and a collaborative 
action research framework (Jensen, Hoagwood, &Trickett, 1999; Telleen & Scott, 2001), the Safe 
Schools/Healthy Students site in Illinois serves the two high schools comprising the J. Sterling Morton 
High School District 201 and its 26 feeder schools. These school districts serve several suburbs directly 
west of the city of Chicago, including Cicero, Berwyn, Lyons and Stickney. Of these, Cicero is the 
largest community, and is comprised primarily of recently immigrated Mexicans. Berwyn, while more 
diverse, is experiencing rapid growth of its Latino population, especially among school age children. 

Method 

Referral 

The Morton area Safe Schools/Healthy Students Initiative provides funding for each elementary, 
middle, and high school to refer two students for community-based intensive case management services. 
Services are delivered by professionals holding a masters degree in social work. Five case managers 
maintain a case load of 12 families each. To be eligible for the program, a student has to be determined to 
be at risk for aggressive and/or violent behaviors by the school social worker or other school mental 
health professionals. Additionally, only students not already receiving services are eligible. 

In the first nine months of the program, 49 students were referred for intensive case management 
and an additional 1 1 students were referred for less intensive services. Of the 49 students who were 
referred for intensive case management, 44 students were still receiving services three months later. Of 
those, 28 students have been receiving services for six months. After 18 months of program operation, 
117 students and their families had received services. 

Most children referred for services are in grades 2-3, grades 6-8, or in their freshman year of high 
school. (See Table 1.) Thirty-eight percent of the students are female and 75% of all referrals are from 
Latino families. English and Spanish is spoken in 27% of the homes; 38% of families speak English 
only, and 35% only speak Spanish in the home. Most children live with both parents (40%) or with 
the mother only (43%); the rest live with their father, a relative or a guardian. 

The children served by Safe Schools/Healthy Students wraparound case management program 
attend school regularly. However, most of the children are doing poorly in school — only 22% perform 
at average for their grade level. Twenty-seven percent of the students have been placed in special 
education classes. 

Each child referred faces multiple risk factors for poor outcomes. The most frequent risk factors are 
academic problems (62%), social isolation of the child (52%), social isolation of the family (35%), 
family violence (40%), drug abuse (32%), alcohol abuse (25%) and unemployed parent(s) (40%). 

Assessment 

Counseling with the child involved evaluating the child at intake using the Child and Adolescent 
Functional Assessment Scale (CAFAS; Hodges, 1994) to assess impairment by determining the negative 
effect of problem behaviors and symptoms on child functioning. The CAFAS subscales assess the child’s role 
performance in school, at home and in the community; behavior toward others; moods and self-harmful 
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Table 1 

Demographic Characteristics of Youth and Family (Afc117) 



Variables 


Percentage 


Place of First Contact 


Family’s Home 


70.0 


School 


15.4 


Phone 


6.0 


Office 


7.7 


Child Grade 


Early childhood - Grade 1 


8.8 


2-3 grade 


23.1 


4-5 grade 


14.9 


6-8 grade 


22.5 


9-12 grade 


15.8 


Child Gender 


Female 


38.3 


Male 


61.7 


Child Ethnicity 


Caucasian 


28.2 


Hispanic 


71.8 


Language Used by Child 


Spanish 


12.3 


English 


46.5 


Bilingual (Spanish & English) 


41.2 


Current Educational Placement 


Alternative school 


4.3 


Regular class 


62.1 


Special education 


26.7 


Drop out 


2.6 


Not in school (e.g., graduate or others) 


4.3 


Language Used by Primary Caregiver 


Spanish 


35.0 


English 


37.6 


Bilingual (Spanish & English) 


27.4 


Marital Status of Primary Caregiver 


Never Married 


14.3 


Married 


41.1 


Separated 


12.5 


Divorced 


22.3 


Widowed 


2.7 


Live-in Partner 


7.1 


Annual Family Income 


Less than $5,000 


9.9 


$5,000-9,999 


17.1 


$10,000-$ 14,999 


19.8 


$15,000-$24,999 


21.6 


$25,000-$34,999 


18.0 


$35,000-$49,999 


9.0 


$50,000-$74,000 


3.6 


$75,000-$99,999 


.9 



behavior; substance abuse; thinking and 
cognition. CAFAS also assesses the caregiver by 
determining the family’s material needs, level of 
social support, and family composition. Based 
on the assessment, a family wraparound plan is 
developed which determines the case 
management and referral plan. 

Case Management 

Using the case management time, activity, 
and referral log developed byTelleen (1999), 
case managers record the referrals made for 
each child and the time spent in various 
aspects of case management. This provides for 
the measurement of the intensity of the 
intervention and of each component of 
wraparound case management. 

One goal of case management is to 
stabilize the families so that they can benefit 
more effectively from mental health 
counseling for the child. This involves 
helping the family find livable, affordable 
housing and transportation to appointments; 
access to food pantries, Women, Infants and 
Children (WIC) and other food programs; 
and to receive employment counseling and 
services for the parent. Family counseling is 
also provided, which helps parents develop 
coping strategies in the areas of parenting and 
life skills. In an effort to stabilize the families, 
case managers access a total of over 25 
different services. This necessitates an 
enormous commitment on the part of the 
case managers, and includes locating 
appropriate resources for families, following 
up to ensure that the families receive the 
services, and maintaining contact with the 
individuals and agencies providing services to 
the family. This wraparound case 
management approach is based on programs 
described by Evans and Armstrong (2002) 
and Burchard, Burns and Burchard (2002). 
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Results 

The longitudinal effects of the wraparound case management intervention were assessed by 
comparing the child’s functioning (as measured by CAFAS) at intake, three months and six months. 
Using a paired t-tcst analysis, results indicated that there was significant improvement in the overall 
functioning of the child for those families who remained with the wraparound case management for 
six months, t (43) = 2.86, p = .007. This significant improvement in child functioning occurred in the 
areas of school functioning, t (42) =2.87, p = .006) and mood/emotions, t (38) = 2.89, p - .006. 

The intensity of the wraparound case management effort was measured using the Telleen case 
management information system developed for this project and based on previous work by Telleen 
(1999). The case managers completed a daily activity log indicating each client seen and the time 
spent on each activity. Over the first ten months of the intervention, the case managers spent time 
assessing the child’s emotional well-being and functioning, assessing the family’s area(s) of need, and 
developing the wraparound plan. After intensive assessment of the family’s situation, the case managers 
spent as much time counseling the family as a whole as they do counseling individual family members. 
Though most of the case manager’s direct service time was spent in counseling (62%), the second 
largest amount of time (38%) was spent in collateral contact with others in the child’s life (e.g., the 
child’s teacher) on behalf of the child or the family, and in contact with family members other than the 
child or caregiver. 

The most prevalent referral activities included visiting the provider with the client and making calls 
to the referral agencies and providers. Follow up with both the client/family and the provider were also 
conducted by the case managers. Eleven percent of the case managers’ time was spent in staffing and 
consultation about the case and direct supervision regarding the case. These activities all necessitated a 
tremendous amount of travel for the case manager. Family counseling was intended to help parents 
develop coping strategies in the areas of parenting and life skills. 

Once the child and family leave the wraparound case management of the Safe Schools/Healthy 
Students Initiative, it is hoped that the family will remain connected to the services to which they have 
been referred. We will contact these families again at 12 months follow up. 

Discussion 

The model of coordination among the five community mental health agencies, as well as the 
coordination of the mental health agencies with school social workers and school psychologists, 
positively affects the retention of families in the service delivery system. In addition, this model of 
coordination positively affects the child’s academic and school functioning as well as social and 
emotional functioning. Other studies have found that parenting behaviors and attitudes are 
significantly improved with the presence of social support, similar to the types of support offered 
through wraparound case management (Andresen &c Telleen, 1992). The wraparound case 
management approach within this Safe Schools/Healthy Students Initiative has had the effect of 
connecting schools, families and communities in providing mental health services for children at risk 
of poor outcomes. (Taylor &C Adelman, 2000; Zins, 1997). 
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Results of a Longitudinal School-Based Intervention Study 

Michael Boroughs, Oliver T. Massey, Kathleen H. Armstrong 

Introduction 

In recent years, violence in the nations schools has become a central concern to society. Media reports 
imply that this is a growing problem that must be addressed. One strategy to combat youth violence in 
schools is intervention via programs to curb violence, reduce substance use and increase safety. 

The Safe Schools/Healthy Students Initiative (SS/HSI) is a U.S. government grant awarded to 
local school districts to fund programs to improve school safety. Pinellas is one of four districts in the 
state of Florida to receive a three-year grant from the SS/HSI. Pinellas County has approximately one 
million residents and a large urban school district with about 111,000 students. Part of the Initiative 
includes an evaluation to measure and report the success or shortcomings of the programs funded by 
the grant. Our role as evaluators is to collaborate with the district, and community agencies that 
provide services, in an effort to collect data using multiple methodologies to report an accurate 
reflection of the grant funded activities. 

One source of data includes referral records that were provided by the district. Each occurrence of 
inappropriate student behavior is collected and organized into a large annual database of individual 
students referral histories. Behavioral categories range from not severe (e.g., tardiness) to severe (e.g., 
battery on a student). While only a minority of students are represented in these files, a disproportionate 
number (i.e., the vast majority of total referrals) are categorized as not severe and non-violent. 

Data have been collected for four years, or eight semesters, and will be analyzed to highlight 
differences in student behavior. While two years of data were collected prior to the beginning of the 
SS/HSI grant (during 1998 and 1999), data from years 2000 and 2001 were collected during the first 
two years of the grant. 
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Although there are over 3 6 different disciplinary referral categories used in this school district, we 
organized the data to be analyzed into two critical clusters: (a) the policy-related or manadatory 
suspension catergory, and (b) the violent referrals category, in order to provide a precise glimpse of the 
more severe disciplinary problems in this school system. The policy group included referral data for 
disciplinary offenses that result in a mandatory suspension from school (i.e., zero tolerance policy). 
Examples of referrals in this category include substance abuse or use or possession of a weapon in 
school. The violence group included referral data for acts such as fighting, sexual harassment or threats 
and intimidation. 

Data on referral histories for policy and violent behavior resulting in disciplinary referrals were 
analyzed. This article presents general trend analyses of disciplinary referrals and discusses these trends 
in relation to the SS/HS Initiative. Five targeted interventions that have been implemented in Pinellas 
County schools will also be discussed. These programs cover interventions ranging from an elementary 
school program supporting parental involvement in the learning process to a high school program that 
offers an alternative to out-of-school suspension. 

Method 

Participants 

This study used secondary data analysis. Using the 2000 school year as an example, 109,628 
students were enrolled in the district during this period with 42,613 students having at least one 
referral. A total of 180,912 referrals were reported in the district during this school year (see Table 1). 
As the data demonstrate, a small percentage of students generate a high number of referrals. That is, 
they are repeat offenders with multiple referrals. 

Analysis 

The most pressing issues are those related to violence and policy referrals and therefore, these 
categories were the focus of our analyses. Comparative percentages and descriptive statistics were used 
to highlight the data. Data analyses were also conducted on the five targeted programs. 

Results 

Analyses of the data over a four year period revealed several trends. For instance, contrary to the 
perception of increased violence and substance use in the schools as reported by media, referrals were 
found to be on the decline in this district. During the 1998 school year, a total of 182, 488 referrals 
were reported. For 1999, this increased to 186, 717 and then began a downward trend to 180, 912 in 
2000 and 139, 339 in 2001. There was a corresponding trend found in aggregregated referrals (i.e., 
unduplicated referrals) which is a measure of the total number of students who received at least one 
referral during a given school year. In 1998, unduplicated referrals were 50, 478; this increased to 58, 
862 in 1999 and then decreased to 48, 828 in 2000 and 47, 914 in 2001 (see Table 1.). 

Only a small percentage of theses referrals were either violent or policy-related referrals. Therefore, 
while most students in Pinellas County never receive a disciplinary referral, other students receive 
many. In fact, the classroom behavior and campus/school rules categories account for over 85% of the 
total referrals in each of the four years analyzed. With bus misconduct added to the mix, these three less 
severe categories account for over 92% of total referrals. Thus, less than 8% of referrals in the district 
included harm to the self or others , carrying a weapon or using or possessing an illegal substance over the 
four year period. Although these results do not suggest that violence and substance abuse are 
negligible, they certainly negate the public perception that violence and policy referrals are in the 
majority, or even a large minority, of referrals. 

Delving further into violence and policy referrals, we examined what had taken place over the four- 
year period of the longitudinal study. When highlighting violence and policy referrals only, some 
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interesting trends were discovered during the period beginning in the fall of 1997 and ending in the 
spring of 2001. 

The percentages of total referrals for the two severe categories are as follows. In 1998, policy referrals 
represented 1.4% of all referrals, followed by 1.3%, 1.4% and 1.2% in 1999, 2000 and 2001, 
respectively. Likewise, in 1998 violence referrals accounted for 3-3% of all referrals, followed by 5.6% in 
1999, 5.3% in 2000, and 6.1% in 2001. In practical terms, there was no change over the four-year 
period. However, it is too early to make any inferential statements from the results presented. 
Furthermore, under 6% of the total referrals over four years were violence related, while under 2% were 
policy related. These two severe categories account for just 8% or less of the total number of referrals. 

During the observed period, referrals for tobacco use decreased by almost one fourth (24.7%) 
while alcohol and weapon referrals remained essentially unchanged. Conversely, drug use rose over the 
four-year period. From 1998 to 2001, drugs referrals increased from about 17% to about 38%. 
According to information obtained from the Safe and Drug Free Schools report (2000), most of this 
increase is accounted for by the use of club drugs (e.g., Ecstasy, MDMA, etc.) or steroid use by males. 

With regard to referrals in the violence category, fighting decreased by over 10%, with an in-kind 
increase in threats and intimidation. The battery category showed a slight decrease over the four years, 
while sexual harassment increased slightly; however, these changes were not statistically significant. 

What is uncategorically the most striking finding in our analyses was the association of school type 
with the percentage of violent referrals. Middle schools had the lowest population of students of all 
non-special schools and yet showed the highest referral rate for violence. In order to examine all 
violent referrals by school type, we took the total number of violent referrals in the district over the 
longitudinal period and broke down just these violent referrals by the type of school. The population 
of middle schools (25,654) is less than half that of high schools (51,053) and yet the percentage of 
violent referrals is more than double across all four years. Table 1 illustrates these findings, with the 
population estimates for each of the four school types and the percentage of the total number of 
violent referrals across the four-year period. 



Table 1 

Violence and Policy Referrals by School Type 





1998 


1999 


2000 


2001 


Elementary Schools 
Population approx. 51,053 


20.0% 


21.8% 


20.8% 


18.6% 


Middle Schools 
Population approx. 25,654 


58.6% 


57.2% 


53.8% 


54.2% 


High Schools 
Population approx. 30,789 


13.3% 


13.0% 


16.0% 


16.6% 


Other Schools 
Population approx. 2,132 


8.1% 


7.9% 


9.3% 


10.6% 


District Census 


108,315 


108,551 


109,628 


111,599 


Unduplicated Referrals 


50,478 


58,862 


48,828 


47,914 


Total Referrals 


182,488 


186,717 


180,912 


159,339 



Targeted Programs 

There were five targeted programs included in the evaluation: Families and Schools Together 
(FAST), Anger Management, Chill Out, Partnership and the On-Campus Intervention Program 
(OCIP). At the elementary level, FAST is a program that supports parents and families with school 
involvement, while partnership is a mental health services program for younger children. At the 
middle school level, Chill Out provides an antiviolence and substance abuse reduction curriculum. At 
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the high school level, Anger Management provides an antiviolence curriculum; OCIP, an alternative 
to typical in- and out-of-school suspension, was available to middle and high school students. 

When examining the trend data, there are a few important points to keep in mind. First, the grant- 
funded activities began in the spring of 2000. Therefore, when analyzing these trends, one indicator of 
success of the program is a decrease in average referrals from the 2000 to 2001 school years. With that 
criterion, the anger management and OCIP programs succeeded in the reduction of referrals, while 
the other programs had no significant increases (see Table 2). 

Second, because information was made available by Pinellas Schools for the 1998 and 1999 school 
years, evaluators decided to use this information to augment the trend analysis for the two years prior 
to the grants inception. While in the case of the elementary school programs (FAST and Partnership) 
and the middle school program (Chill Out), the prior years data are easily interpretable; however this 
is not the case for the high school programs (Anger Management and OCIP). As seen in Table 2, the 
trend for these high school programs spikes up and then drops. While it would be wonderful to 
attribute this finding to program effectiveness, it is highly unlikely. The most reasonable explanation 
of the spike would be error in data reporting that has something to do with high school level referrals. 
However, it is important to note in the case of OCIP that there is still a downward trend even if 2000 
figures were not included. That is, without the spike, OCIP participant referrals still dropped over 
time. Anger management on the other hand returned back to the 1999 average in 2001 after the spike. 

Table 2 

Longitudinal Average Referrals by Targeted Program 





1998 


1999 


2000 


2001 


Anger Management 


5.7 


4.9 


12.1 


4.9 


Chill Out 


0.88 


2.2 


4.3 


5.2 


FAST 


0.3 


0.2 


0.6 


1.6 


OCIP 


7.2 


7.4 


11.6 


5.4 


Partnership 


0.3 


0.6 


0.4 


1.9 



Discussion 

The information presented in this longitudinal study is tentative because the grant period has not 
yet expired. Only after all of the data are submitted for analysis, including the 2002 and 2003 school 
years, will we have a clearer picture about what effect these intervention programs may have had on 
trends in disciplinary referrals. A goal for the school district is to both reduce the total number of 
referrals while also reducing the severity of referral types. We have demonstrated, at least initially, that 
this trend is already underway. Total referrals are down while the district census is up, and violent 
referrals remain steady despite increases in the student population. 

Data on referrals show a global outcome, in that they do not tell us how or why behaviors increase 
or decrease; all these data tell us are whether behaviors changed. Therefore, while no concrete 
explanations can be drawn from these analyses, the data contribute to a better understanding of the 
evaluation process and provide insight into possible future trends in student behavior. 

Results show that while violence does occur in Pinellas County schools, the frequency of violence 
appears to be much lower than perceived by the general population. The benefactors of this 
information include parents, especially those with children in the schools; the district, particularly 
administration and professionals who are given an accurate report of what indeed takes place; and 
communities, researchers, and evaluators. 

When these referral categories for violence were analyzed, it was found that shifts in referral 
frequencies had occurred. Within the violent category, both battery and sexual harassment remained 
steady while fighting decreased and threats and intimidation increased. Certainly the goal is to 
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eliminate these behaviors altogether, but it is a positive step that some physical violence is reduced in 
lieu of threats. 

Within the policy category, alcohol and weapons remain constant over the four-year span with 
only negligible increases in these categories. Conversely, a great shift was present with a large reduction 
in the use of tobacco, while the use of other drugs almost doubled. Trends in this category may be due 
to the institution of zero tolerance policies, or the fact that data on drug use increases are clustered, in 
particular, around the use of club drugs such as ecstasy (MDMA) and steroid use in males. 
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Student Perceptions of School Safety in the Omnibus Survey 

Kelli S. Henson, Kathleen H. Armstrong, Oliver T. Massey 

Background 

In 1989, Pinellas County School District began the Omnibus project in order to collect data 
following a cohort of students from kindergarten until they graduated from high school. The project 
began with 8,268 children when they registered for kindergarten in 1989. Survey information was 
collected about the students from parents each year from 1990 to 1998, except for 1996 and from 
teachers each year from 1990 until 1996. Students filled out the Omnibus surveys beginning in 1992, 
from 1994 to 1999, and in 2001. A Pinellas County Schools committee determined survey questions 
each year. Information obtained from Omnibus surveys has been used for several purposes including 
early identification of educational failure, tracking high-risk student progress, nutrition and school 
performance, and child health practices. 

For the 2001 Survey, The Pinellas County Safe Schools, Healthy Students Initiative (SS/HSI) 
evaluation team submitted additional questions about student perceptions of school safety to the 
Omnibus committee for addition to the survey. These questions compliment items on the School 
Safety Survey (Massey, Armstrong, &c Santoro, 2001) developed by the SS/HSI evaluation team to 
examine school staff perceptions of school safety. 

Three sections of school safety questions were added to the 2001 Omnibus (Pinellas County 
Schools, 2001). In section one, students reported the frequency with which they witnessed or 
experienced violent or aggressive events in school. Students were asked to record how often they 
witnessed or experienced 1 1 items on a five-point scale from 1 {never) to 5 {daily). Section one 
included items such as verbal threats in school and students bringing weapons to school. In section two, 
students were asked to rate the effectiveness of violence prevention strategies in their schools. Students 
rated 13 strategies on a five-point scale from 1 {very effective) to 5 {very ineffective). For example, 
students were asked to rate the effectiveness of strategies including suspending/expelling students who 
commit acts of violence and training teachers to resolve conflicts. In section three, students were asked to 
rate their level of agreement with the application of school rules, such as principals apply rules fairly and 
some students are getting away with too much. Students rated their level of agreement with these items 
on a five-point scale ranging from 1 {strongly agree) to 5 {strongly disagree). 
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Methodology 

Participants 

The 2,413 participants consisted of 1,238 females and 1,158 males who attended fifteen Pinellas 
County high schools. Seventy-seven percent of the participants were White, 14 % were Black, 4 % 
Hispanic, 4 % Asian, and American Indian and multi-ethnic participants comprised the remaining 
percent. Eighty-six percent of participants were eleventh grade students, while tenth grade students 
(students who had repeated one grade) made up 12.5% of the sample. Less than 1% of participants 
were in the ninth or twelfth grades. 

Procedure 

All eleventh grade students were given surveys to fill out during school hours. Surveys were also 
distributed to students in the original cohort who had either been retained or advanced a grade since 
beginning kindergarten. Their participation in the study was voluntary; participants were assured that 
the information provided was confidential and would be reported as group data with no identifying 
individual information. 

Results 

Section 7 ; Witnessed or experienced violent or aggressive events in school 

In section one, students reported the frequency with which they witnessed or experienced events in 
school (Table 1). Bullying and teasing were reported most often with over 40% of students witnessing 
or experiencing teasing or bullying on a daily basis. Verbal threats were experienced or witnessed at 
least once per week by 39.5% of students. Additionally, more than 35% of students experienced or 
witnessed discrimination at least once per week. Weapons and gang activity in school were reported 
with low frequency (Table 1). More than 60% of students reported never seeing or taking part in gang 
activity in school. More than 50% of students reported never bringing a weapon or seeing another 
student bring a weapon to school. In addition, 48% of students reported never seeing drugs sold in 
their school. 

Table 1 

Frequency of Witnessing or Experiencing Violent or Aggressive Acts 







% 


% 


% 






% 


1 or 2 times 


1 or 2 times 


1 or 2 times 


% 


ITEMS 


Never 


per year 


per month 


per week 


Daily 


Verbal threats in school 


21.3 


22.0 


17.1 


16.9 


22.6 


Physical violence in school 


18.5 


31.3 


29.2 


13.3 


7.7 


Students bringing weapons 
to school 


57.1 


30.9 


6.4 


1.6 


3.9 


Students using drugs or 
alcohol in school 


29.9 


22.6 


16.4 


11.7 


19.4 


Drugs being sold at school 


48.0 


19.9 


13.1 


9.1 


9.9 


Teasing or bullying in 
school 


14.0 


12.7 


14.1 


16.8 


42.4 


Gang activity in school 


62.5 


18.2 


8.3 


4.4 


6.6 


Personal property stolen or 
destroyed in school 

Vandalism of school 


19.9 

22.4 


31.5 


23.4 


13.5 


11.8 

15.4 


31.8 


20.1 


10.3 


property 






Discrimination at school 


23.2 


23.2 


18.3 


11.8 


23.6 


Violence in the community 
where your school is located 


35.8 


28.0 


18.2 


8.5 


9.5 
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A chi square analysis was used to identify the significance of association between two variables. 
For purposes of this analysis, students were split into two groups for each question in section one: 

(a) students who reported seeing or experiencing an item frequently (once a month or more), and 

(b) students who saw or experienced an item rarely or never (2 times a year or less). 

A chi square analysis of those items and gender revealed that males reported seeing or experiencing 
acts of violence or aggression significantly more often than females reported (yf = 24.756). For 
example, significantly more male students reported seeing or experiencing physical violence, weapons 
being brought to school, gang activity in school, and verbal threats in school than female students. 

A chi square analysis of the items and ethnicity revealed that White students reported witnessing or 
experiencing acts of violence or aggression significantly more often than other students. In particular, 
White students reported witnessing or experiencing significantly higher numbers of verbal threats, 
incidents of teasing and bullying, and physically violent acts in school than other students (yf ~ 25.032). 

Section 2: Effectiveness of violence prevention strategies 

These items were designed to assess student opinion of the effectiveness of school violence 
prevention strategies. Students reported that controlling guns in school (68.5%), having counselors to 
help students in school (53-3%), and keeping drugs out of school (52.4%) would be the most effective 
strategies to help keep their school safe. According to students, putting more security devices in 
schools and training students in conflict resolution and anger management are the least effective 
violence prevention strategies in schools. 

Section 3: School discipline 

In this section, students reported their level of agreement with school disciplinary measures in 
response to behaviors that disrupted safety at school. Answers were normally distributed with a nearly 
equal number of students agreeing and disagreeing with each item. 

Students are expected to complete a final Omnibus survey in May 2002, the expected year of 
graduation for their cohort. The SS/HSI evaluation team has submitted additional items to Section 1: 
Witnessed or experienced violent or aggressive events in school. However, items that were included in 
Section 2: Effectiveness of violence prevention strategies, and 3: School discipline, will be replaced 
with items relating to resilience and protective factors in the school. In addition, the SS/HSI 
evaluation team plans to conduct focus groups with high school seniors during the 2001-2002 school 
year in order to find out more about their experiences of school violence and safety during their school 
years. These focus groups may be able to tease out more detailed information about what students 
think help to keep schools safe. 

Conclusion 

This paper presents results of a survey of high school students who have been followed since 
beginning kindergarten and were due to graduate in 2002. One of the most startling findings was that 
over 40% of students experienced or witnessed teasing or bullying on a daily basis. Controlling guns 
in schools, having more counselors to help students, and keeping drugs out of school were reported by 
over 50% of students as the most effective strategies helping to keep schools safe. 
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Safety in Schools: Staff Perceptions 

Gina Santoro, Kathleen H. Armstrong, & Oliver T. Massey 

Introduction 

There is a recent trend whereby youth are becoming increasingly involved as perpetrators and 
victims of violent crimes. Youth aged 18-24 comprise the highest rate of violent offenders and victims 
of violent crimes (Fox & Zawitz, 2000). Overall, the average age of both victims and offenders of 
violent crime has been declining since the late 1980s. Young people are three times more likely than 
adults to be victims of violence, and one-fourth of these victimizations involve the use of a firearm 
(Arredondo, et al., 1999). Although violent crime rates have decreased overall, crimes have become 
more intense when they do occur (Anderson, et al., 2001). Additionally, while single-victim events are 
decreasing, multi-victim events are increasing. As a result, when accounting for the increasing intensity 
of the multiple-victim events, the total homicide rate for students killed at school actually has 
increased in recent years. The proportion of school-based deaths involving multiple victims increased 
from 0% in 1992 to 42% in 1999 nationwide (Furlong & Morrison, 2000). Acts of targeted school- 
based violence are extremely rare, but when they occur, they hamper student development, reduce 
academic learning and harm the schools climate (Furlong & Morrison, 2000). 

Studying the perceptions of school safety from those most intimately affected is one avenue for 
investigating the influence of school violence on school climate. This paper discusses the results from a 
survey intended to identify educators’ perceptions of safety in a large urban school district, which had 
received federal funding from the Safe Schools/Healthy Students Initiative (SS/HSI). Implications 
from these findings may be useful to both educators and program evaluators. For educators, such 
descriptive information may be used as a needs assessment at the school and district level to identify 
areas of need in order to allocate resources for appropriate interventions. For program evaluators, this 
information can be used for accountability and tracking purposes in evaluating the efficacy of the 
interventions. 

Method 

In order to investigate staff perceptions of school safety, the School Safety Survey (Massey, 
Armstrong, & Santoro, 2000) rating scale was developed by the evaluation team and administered to 
staff members (e.g., teachers, counselors, administrators) in Pinellas County, Florida. The instrument 
includes 36 items that sample six factors associated with school safety: administrative effectiveness, 
student crime, child behavior problems, family involvement, teacher effectiveness, and school location. 
Respondents were asked to rate “How serious are these issues at your school?” using a multi-point scale 
ranging from 1 (extreme problem) to 5 (not a problem). Questions related to administrative effectiveness 
included items about effective leadership and disciplinary practices and procedures. Student crime 
included items about illegal activities at school, such as selling or using drugs and alcohol. Child 
behavior problems included items such as teasing and bullying of other students. Family involvement 
included items about disciplinary practices and school involvement. Teacher effectiveness included items 
regarding supervision of students in the classroom and classroom management practices. School 
location included items that inquired about the safety of the area surrounding the school. 

Participants 

Administrators from eight schools and the district counseling staff agreed to participate in the data 
collection efforts. Nine hundred instruments were distributed to staff at three high schools, three 
middle schools, two elementary schools, and the district counseling staff. Four hundred seventy-seven 
completed instruments were returned for a 53% response rate. Forty-three percent of the instruments 
were completed by high school staff, 27% by middle school staff, and 28% by elementary school staff. 
Sixty-two percent of the sample was made up of teachers, 26% were school counselors, 2.5% were 
administrators, and 6.5% held other positions within the school. 
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Results 

Concerns were examined separately for elementary, middle, and high schools. Insufficient parental 
involvement and support in addressing discipline were the main concerns at the elementary school 
level, with 49% of respondents rating it as a serious or extreme problem. Lack of alternatives to 
suspension and expulsion (36%), teasing among students (36%), ineffective disciplinary practices 
(33%), and verbal threats among students (33%) were other serious concerns for elementary school 
respondents. Middle school participants reported that teasing among students was their most serious 
concern, with 39% of respondents rating it as a serious or extreme problem. Other serious problems 
reported at the middle school level were bullying among students (34%), verbal threats among 
students (31%), and lack of alternatives to suspension and expulsion (28%). High school participants 
indicated insufficient parental support in addressing discipline problems as their greatest concern, with 
39% of respondents reporting it as a serious or extreme problem. Other serious problems reported at 
the high school level included insufficient parental involvement in school (35%), inadequate 
supervision of students during class (32%), and teasing among students (29%). 

Teasing among students emerged as a significant concern at all levels: elementary, middle, and high 
schools. Both high schools and elementary schools reported that reduced parental support created a 
serious concern. Elementary and middle schools agreed that administrative policies and practices were 
related to more serious problems at school, while decreased classroom supervision was problematic at 
high schools. 

Conclusion 

These data indicate that while there are some common concerns regarding the perception of safety 
at schools, there are also differences that also need to be addressed according to the level of the school 
and the age of the students. Insufficient adult involvement and supervision of child behaviors, whether 
parents, teachers, or administrators, was identified as compromising the sense of school safety at all 
levels. With respect to intervention planning, schools will need to carefully determine how to best 
promote parental involvement — a goal that will probably require more effort at different levels. For 
example, parents of younger children may become more easily involved than parents of middle school 
students. Problematic student behaviors, especially teasing and bullying, must be attended to at all 
school levels, and both victims and bullies must be taught acceptable social and problem solving skills 
early on and throughout school. Finally, alternative consequences for more serious misbehaviors must 
be developed, rather than the current and often ineffective practice of suspension and expulsion. 
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Understanding School Safety: A Qualitative Inquiry 

Angela Perry, Michael Boroughs, Kathleen H. Armstrong, & Oliver T. Massey 

Violence in the schools is a very complex problem. Despite several recent high-profile violent acts 
in the schools, much of the violence in schools is not life-threatening and incidents have actually 
decreased over the past eight years (Elliot, 2001). Nevertheless, the public has been inundated with 
news of school violence in the media. School shootings perpetrated by classmates have gained an 
enormous amount of publicity. An Interim Report by the U.S. Secret Service and Department of 
Education (2000) stated that such publicity focuses on rare events and has resulted in increased fear 
among students, parents, and educators. While school shootings are serious, they are actually 
uncommon events and, in fact, schools are one of the safest places for students to be (Vossekuil, 

Reddy, Fein, Borum, & Modzeleski, 2000; U.S. Department of Education, 2000). 

A myriad of programs have emerged in schools to prevent violence. In an effort to document the 
issues surrounding school safety, the Safe Schools/Healthy Students Initiative (SS/HSI) evaluation 
team conducted a series of focus groups with students, parents, and teachers in the Pinellas County 
schools. The goal of these focus groups was to gain an understanding of the experience of safety in the 
schools, from multiple perspectives. While national data show that schools are indeed safe places, our 
aim was to qualitatively measure how these groups felt about safety in their schools. Qualitative 
analyses of transcripts from these groups revealed some interesting findings that in some ways support, 
while in other ways contradict common perceptions about school violence and safety. 



Method 

Participants 

Eight focus groups were conducted in Pinellas 
County over the 2000-2001 school year, and 
included a total of 70 participants who were placed 
in student (n = 37, including elementary and middle 
school students), parent (n = 21, including parents 
of elementary, middle and high school students), or 
educator {n - 12, including teachers from 
elementary schools) groups. Table 1 describes 
demographic characteristics of the participants. 
Participation was voluntary for all participants and 
responses were confidential. 

Procedure 



Table 1 

Demographic Characteristics of 
Focus Group Participants (/V= 70) 



Demographics 


Students 

(N=37) 


Parents 

(N=21) 


Educators 

(N=J2) 


Age Range 


9-15 


NA 


NA 


Grade Range 


3-8 


NA 


NA 


Gender 


Male 


25 


NA 


12 


Female 


12 


NA 


0 


Race 


White 


10 


16 


9 


Black 


3 


4 


2 


Other 


24 


1 


1 



Potential student participants were identified via faculty and staff referrals and parental consent was 
granted in advance for all students that participated in the study. Parents were recruited through the 
Student Advisory Council Teams (SAC) at two of the schools and all parents were given the 
opportunity to participate at a third school. Focus groups lasted 43 minutes for students and ranged 
from 1 to 1.5 hours for parents and teachers. The focus group data were recorded using two audio 
recording devices, then transcribed and finally coded and analyzed using Ethnograph software (Qualis 
Research, 1998). 



242 - Research and Training Center for Childrens Mental Health - Tampa , FL - 2003 



16 



Symposium: Evaluating Program Effectiveness: Safe Schools Healthy Students Initiative 



Results 

Student Responses 

Notable differences emerged between elementary and middle school students with regard to feeling 
safe at school. Elementary students reported a fear of intrusion or danger from the outside. This fear 
varied based on the school structure (open versus closed campus) and safety drills (i.e., lockdown 
procedures for intruders on campus). While attending school, elementary students reported feeling 
relatively safe. However, they reported concerns about fighting and the use of profanity within the 
school. Their greatest fear related to threats from others outside of the school. Elementary students 
reported that they preferred being in school rather than at home because there were more activities in 
school. Middle school students, by contrast, worried more about threats from other students. Bullying 
and teasing were identified as major problems in the schools and contributed to violent outbreaks: 

Like you dont know what th eyre gonna do and thats why a lot of shootings happen 
because a lot of people pick on other people and they just get sick and tired of it and 
fed up. And they dont know what to do to make it stop. (Middle School Student) 

Transitioning to the next level of schooling (from middle to high school) created anxiety according 
to our student participants. Worried about moving from middle school to high school, middle school 
students expressed the need to really “watch yourself” in order to stay safe. Further, middle school 
students worried about being shot or getting caught up in gang activity. 

Both elementary and middle school students reported that teacher attitudes and practices contributed 
to their decreased sense of safety. Both groups commented on “yelling teachers,” whose moods 
contributed to uneasiness in the classroom. Participants attributed student conflict, in part, to a lack of 
adequate discipline by teachers. Middle school students portrayed teachers who failed to properly deal 
with conflict, especially when a student was consistently being bullied or teased, as follows: 

They didn't handle like how they were really supposed to. And at the end something 
happened that was really dangerous and it could have gotten worse. (Middle School Student). 

Elementary and middle school students alike stressed concern about the lack of supervision at 
school, and overwhelmingly reported feeling safer when they had more adult supervision. Elementary 
students wanted supervision for the purpose of being protected against intruders, while middle school 
students wanted adults to address conflict when it arose among students. Younger children (grades 3 
and 4) reported that attentive teachers could help promote a feeling of safety among students. Fifth 
graders and middle school students acknowledged the role of policies and procedures in their feelings 
of safety. The student council at the fifth grade level was one way of dealing with problems within the 
school while enforcing rules to prevent further problems. Middle school students acknowledged that 
policies and procedures were a good idea, but were not convinced that they were serving their purpose. 
As one student stated, “Policies don’t make me feel safe at all because just cause its a policy doesn’t 
mean people are gonna do it.” 

There was also consensus across the groups that parents contribute to their child’s sense of safety 
when they visit them at school and follow up on any conflicts their child may be involved in. 

However, parents also contribute to a decreased sense of safety when they fail to discipline their own 
children or fail to encourage appropriate behavior in school. 

Parent Responses 

Parents of elementary, middle and high school students shared the perception that school is less 
safe now than when they were growing up. Specifically, parents of middle and high school students 
were concerned about drug and gang activity, weapons, and bullies. Parents of elementary and middle 
school students reported a fear of their children transitioning to the next level (i.e. from elementary to 
middle school or middle to high school). 
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Parents were concerned with safety issues within the schools, outside of the schools, or from 
unknown factors. Concerns about the location of schools also surfaced in these focus groups. Parents 
felt that dangers existed in the neighborhood, and that neighborhoods were less safe than schools. 
Neighborhood characteristics such as high drug/crime areas, racial tensions, and lack of supervision at 
the bus stop were perceived to create dangerous situations for students outside of school. 

Many parents reported dissatisfaction over the dearth of information being provided to them 
regarding school problems. Parents consistently blamed other parents for problems affecting schools. 
Most often they attributed the problems to other parents’ lack of supervision and discipline of their 
children, modeling of bad behavior, and allowing children access to weapons at home. One group of 
parents blamed the government for laws and regulations that “tie the parents’ hands” and do not allow 
them to control their children “like they need to.” 

Teacher Responses 

Teachers’ concerns mirrored those of students, in that they felt relatively safe in their classrooms 
but feared intrusions from the outside. Teachers reported that disrespectful and sometimes hostile 
parents, the size of the school facility (too small), the location of the school and open campuses 
contributed to the vulnerability of problems. While policies and procedures are in place to assure 
safety, these teachers reported that the procedures themselves rendered feelings of unease. The constant 
drills to prepare children for threats seemed to be causing fearful reactions and would possibly 
desensitize them to dangerous events. 

Teachers blamed parents for doing a poor job of modeling appropriate behavior for their children. 
There were stories of parents who were disrespectful to teachers in front of the students, not involved 
in the lives of their children, and provided insufficient supervision at home. Teachers reported that 
student behavioral problems were a result of parents’ failure to reinforce rules and expectations taught 
at school. 

Conclusion 

The findings from these focus groups suggest that students, parents, and educators believe that 
while schools may be safe, a number of issues remain that compromise safety. Teasing and bullying 
were identified as two behaviors that make school unpleasant, if not horrific, for some students. The 
interschool transitions, especially from elementary to middle school, created feelings of dread for 
students and their families. To address these issues, both parents and teachers must take responsibility 
for student supervision and discipline, and schools need to do a better job in preparing students for 
transitions. A review of the procedures being implemented in schools to keep students safe needs to 
occur so that undo anxiety is not created among students and staff. 

While statistics may show that school is one of the safest places for students to be (Vossekuil, et al., 
2000; U.S. Department of Education, 2000), this perception, or sense of safety, has been 
compromised by a combination of real events and the repetitive reporting of them in the media. If 
students and teachers are uncomfortable in school, learning will be compromised. It is essential that 
schools and communities come together to discuss these issues, identify concerns, and develop 
interventions that will successfully address and overcome the belief that schools are unsafe places. 
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Symposium Discussion 

Kathleen H. Armstrong 

The Safe Schools/Healthy Students Initiative was offered to school districts across the nation by 
the federal government, to create an array of prevention and intervention efforts to stem the tide of 
violence in our schools and among our youth. These efforts were to include community providers, 
specifically mental health and juvenile justice, in a proactive and collaborative manner, in order to 
expand and strengthen supports and services needed for healthy child development. While most of the 
evidence points to schools as being safer than homes and communities, the question remains as to how 
best to create a safe and healthy school. Each of the papers presented in this symposium help us to 
identify key areas that must be addressed if we are to achieve that goal. 

Creating a positive school climate is essential as a foundation for a safe and healthy school. This 
means that school must be a positive place for both students and teachers. Students must have 
opportunities to learn both the academic and social competencies required for success in school. 
Teachers must be able to teach in an orderly classroom environment, free from distractions that often 
stem from disruptive behaviors and conflicts. Administrators must provide teachers with the support 
that they need to meet the needs of an ever-growing diversity of learners. And parents must support 
schools, by setting high expectations for their childrens education and behavior. 

To make this happen, it will be necessary to create an array of interventions in schools that are 
multifaceted and start early on in a child’s development; this will promote social and academic 
competencies in all students. No program by itself is likely to be successful in these efforts; rather the 
combined efforts of schools, families and communities will be necessary. Strengthening families, 
especially young families just starting out, will be a key factor toward achieving the goal of safe and 
healthy schools. Providing high quality childcare and preschool programs is another factor that promotes 
healthy child development and success in school, even for children living under adverse circumstances. 
Teaching not only academic skills, but also social and problem solving skills equips children with the 
social and emotional competencies that are of critical importance to school success. Providing a 
classroom with clear and consistent rules, fair discipline, and high expectations for all children creates a 
positive and supportive classroom culture. Strong and effective leadership in schools creates security and 
promotes high quality education. In addition, the willingness of leaders in the community to collaborate 
and share resources promotes not only the health of schools but the community at large. 

And, finally, we need to pay more attention to transitions, those times in human development 
when we move ahead to a new, and often-unfamiliar environment. Perhaps one of the most vulnerable 
times for students involves the transition from elementary school to middle school, which happens at 
the same time that children become young adolescents. At a time in life when it is most important to 
feel a sense of belonging, adolescents leave the more protected world of elementary school and move 
into the world of middle school, a more diverse, more competitive, and often less supportive place. 
Teasing and bullying reaches its peak in middle school, and for vulnerable students, this period may 
begin the decline of a successful school career. Coupled with physical development and school changes 
comes the increased risks from engaging in unprotected or early sex, drugs and alcohol. Eating issues, 
resulting in obesity or eating disorders such as anorexia and bulimia often emerge at this time. 
Television watching, computer surfing, or electronic game playing often fill up the young teen’s 
unsupervised hours, and may increase isolation or feelings of despair. 
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It is time that communities, schools, and families come together to create structures that promote 
healthy child development. For those children at-risk, schools may serve as the most influential 
protective factor available. In order to be effective in this very important role of providing children 
with the tools needed for success, communities and families must also support school efforts. And for 
children and families requiring additional support, there needs to be an array of interventions readily 
available in the community. Only when those critical pieces are in place will we reach the goal of 
creating safe and healthy schools. 
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